Joshua Hull

Building Hours

Sun, 15 Mar 2020 21:00:00 +0000

Angel Flight Soars

Civil Air Patrol

Pilot's License

Sun, 15 Mar 2020 20:00:00 +0000

Private Pilot's License

King's School Private Pilot Syllabus

Clemson University Flying Club

Checkpoint	Hours	Approx. Cost¹
Familiarization and Basic Control	6.5	$975
Refining Control and Learning to Land	6.7	$990
Expanding Maneuvers and Landings Skills	5.9	$730
Night and Cross Country	14.7	$1940
Earning your certificate	6.4	$880
Total	40.2	$5515

Instrument Rating

King's School Instrument Rating Syllabus

Checkpoint	Hours	Approx. Cost²
Learning and Refining Aircraft Control Using the Instruments	8.8	$1400
Navigating While Flying on Instruments	9.3	$1500
Finding the Airport – Flying Instrument Approaches	11	$1800
Instrument Cross Countries	6.7	$1000
Becoming Instrument Rated	4.2	$670
Total	40	$6400

Instruction @ $50/hour, Aircraft @ $100/hour ↩︎
Instruction @ $50/hour, Aircraft @ $110/hour ↩︎

re:Invent 2018 Reactions

Wed, 28 Nov 2018 10:49:17 -0500

I've been looking on a good topic to post about before I can get into more example posts and with re:Invent going on this week I thought it might be worthwhile talking about a few of the announcements I'm excited about.

AWS Transfer for SFTP

A lot of the data that I handle at work is given to us by vendors/partners as well as clients. While I would love to have them be able to drop the data into S3 a lot of tools and processes revolve around older protocols like SFTP. With AWS offering a managed SFTP server this will be a good bridge between the old world and the new one. I particularly like that the service integrates with external authentication like LDAP and Active Directory. This will allow not only other parties to handle provisioning the accounts but password reset build around those authentication resources.

EC2 Instances (A1) Powered by Arm-Based AWS Graviton Processors

Back when I first started college I began my academic career studying Electrical Engineering. Ever since then I've enjoyed tinkering with electronics whenever I can. Part of the joy of tinkering has meant a fondness for ARM processors. To me, they always seemed so approachable as someone who's dreamt of building their own computer for so many years. When I heard that AWS had announced the a1 instance family I was very excited. Not only because of that fondness for them but for more practical reasons as well. Things like ARM docker images being built by GitLab, integration testing on Android-based instances, and others make me excited for the future of this and other ARM instance families.

As a side note, I was able to spin up a Docker Swarm cluster on a1 instanced within 24 hours of their being announced. I'm hoping to have a blog post about that ready shortly that also discusses some other ideas and a few I've mentioned here more in depth.

Firecracker – Lightweight Virtualization for Serverless Computing

This particular announcement I'm most excited about because I'm excited to see what the community with do with it. I can envision local versions of Lambda being developed as well as the project being integrated into other projects or services.

AWS Ground Station

Anyone who follows me on twitter (you do follow me, don't you?) knows that I'm a big fan of everything New Space/Space 2.0/what ever you want to call it. If anyone is looking for someone to pay to build and launch a CubeSat feel free to drop me a line. Anyway, this announcement to me means that a big player in the cloud space is ready to take New Space seriously. I've seen several people wonder why this service would be offered. To me, it doesn't seem all that surprising. Jeff Bezos is personally funding a rocket company. In addition, I know of one other company that is leveraging AWS and had modeled their process in a similar way. My only hope is that AWS takes “cloud” computing literally and we see a new region announced: leo-1

Amazon FSx for Windows File Server

Like I mentioned in the section about AWS Transfer for SFTP a lot of the workloads I see on a day to day basis are based off older technologies. I've also seen many based on Windows technologies (I'm more of a Linux fan/user). Having a Windows-based file share means being able to more easily interface with those processes that aren't able to adapt to new AWS-centric technologies like S3.

Control Tower

As someone who's spent two and a half years working on the AWS platform, I think I can say this with a level of confidence: Best practices are hard. Setting everything up, monitoring, all of the various tasks. The fact that AWS is giving an out of the box way to accomplish this hopefully will lead to more and more of these best practices being adopted and all the benefits associated with them reaped.

Amazon Textract

Recently here at work, there was a discussion about the possibility of using AWS’ Machine Learning technologies to extract information from forms that would be sent to us by customers. We ended up going in a different direction but the idea of being able to scan documents and extract the results got me thinking. I personally would use it to take care of receipts and things like that. With Textract all of that is an S3 bucket, a Lambda function, and a scanner away. Receipts, forms, bills, a plethora of items could be scanned, organized, and tagged.

AWS Outposts

With all these posts about AWS, it might surprise people to know that I'm also a sucker for physical servers too. There's just something about having a big, loud, power hungry stack of computers that lets you know work is getting done. I've always secretly wanted to run my own personal cloud based either out of my home or a local data center when the power bill gets too high. Projects like Openstack have meant that that was a real possibility. With the announcement of Outposts, this dream of running a cloud locally is even more possible! In reality, I can see this being used in a lot of places. Manufacturing, Education, the list goes on. My only concern is how much is it going to cost for me to have us-joshs-cloud be a reality.

AWS Managed Blockchain

I've had an issue in GitLab for the blog for a post about developing a CI/CD pipeline for Ethereum contracts. One of the biggest headaches I've been trying to wrap my head around has been how to deploy to the production network and public test networks. Testing is easier thanks to the folks who developed Ganache but the public networks are a different story. I've debated docker containers and AMIs and how to keep the startup time for them down to a minimum and other complications. However, with AWS Managed Blockchain I expect a lot of those headaches to go away. With this service, I'm hoping that AWS removes a lot of the operational headaches of running an Ethereum node and allow for more rapid development on the platform.

AWS Lake Formation

Earlier this year we undertook an exploratory dive into migrating our process from its current state to that of a modern data architecture. This involved the setup of a data lake and exploring various ETL tools. While our particular use case we were exploring didn't quite fit I can say that the concept definitely stuck with me. What also stuck with me was the complexity of such an undertaking. Based on what I saw with Lake Formation I'm eager to see what headway AWS has made into simplifying the process of moving to a data lake strategy.

This is turning out to be the longest post I've written for any of the iterations of this blog so I'm going to leave it there for now. I'll write the rest of my reactions in part two to be released shortly.

CI/CD With Cloudformation

Mon, 10 Sep 2018 11:03:43 -0400

Some astute reads may have noticed a design change around this blog and you would be correct. I've migrated it over to the awesome Hugo engine and the Academic Theme. All of this was of course facilitated by the choice to migrate over to GitLab. There were a couple of reasons that I decided to move over but the major one was being able to have CI/CD directly tied into my repos. I'm a big proponent of CI/CD and DevOps, so much so that it's one of my roles at my full time jobs in addition to all the AWS work I do. How those two roles intersect brings us to the motivation behind this post. I've started using CloudFormation, AWS’ Infrastructure as Code offering, more and more at work. The ability to reliably reproduce work, change track it, and deploy it multiple times is quite simple awesome. When I decided to make the shift over to GitLab I chose CloudFormation as the first task to use GitLab's CI/CD features, other then this blog of course. I have to say that is was remarkably straight forward to set up. I've created a repo over at https://gitlab.com/joshua_hull/cfn-templates that includes some working examples of templates as well as the .gitlab-ci.yml that I use for the project. As I do more work with CloudFormation I hope to add more the project but for now it serves as a good example for anyone wanting to explore CI/CD and CloudFormation on GitLab.

Using Scheduled Lambda Functions To Snapshot Volumes

Mon, 10 Sep 2018 10:22:40 -0400

One of the primary methods I use for backups when working inside AWS is the use of snapshots. They're straight forward in AWS and they're available in the AWS CLI which means they are scriptable. In the past I'd write a simple BASH script and schedule it with Cron but this is the year of Serverless so I decided to re-write the process with AWS Lambda. While I'm not going to post the exact code I'll cover a few of the tidbits below.

The premise of the script is that it will run on some schedule and describe all volumes in the account it's operating under. It will then filter out volumes that don't contain at least one tag that matches a regular expression. This RegEx will make sure we're working on volumes that contain the schedule the volume should have a snapshot taken on, the last time it was run, the name of the snapshot, etc.

Given my familiarity with the JavaScript AWS SDK and NodeJS in general I chose to use that framework to develop the lambda script. Since the JS SDK can return Promises I tend to make these types of lambdas a chain of promises, similar to this:


params = { ... }
EC2.describeVolumes(params).promise().then(data => {
    let promises = []
    // Magic goes here.
    return Promise.all(promises);
})
.then (data => {
    let promises = []
    // Even more magic goes here.
    return Promise.all(promises);
})
.then(data => {
    callback(null, data);
})
.catch(err => {
    callback(err);
})

Fill in the blanks and you have a pretty handy Lambda function that can work with not only the EC2 API but any AWS service that returns promises through the JS SDK. One example is the SNS service. I use this one to send out emails when the snapshot has been preformed but you could just as easily send them out in the catch() to report errors instead of success.

One big thing to emphasize though is that most of the configuration happens on the volumes themselves. In my particular use case the volumes have snapshots created varying schedules. This requires setting up multiple cron jobs or lambdas with different configurations. However, with this new lambda each volume can have its own configuration that can be updated with me having to re-deploy the lambda with a new configuration itself.

Lambda Function and Encrypted S3

Fri, 05 May 2017 01:09:22 -0400

One of the aspects of AWS Lambda¹ that makes it excepent is that Lambda is used to extend other services offered by AWS. In this example we will set up Lambda to use Server Side Encryption for any object uploaded to AWS S3².

The first task we have is to write the lambda function. Below we have the Python code that will read in the metadata about the object that was uploaded and copy it to the same path in the same S3 bucket if SSE is not enabled.


import boto3
import os
import sys
import uuid

def check_if_unencrypted(bucket, key):
    s3 = boto3.resource('s3')
    obj = s3.Object('bucket_name','key')

    return not obj.server_side_encryption


def handler(event, context):
    s3 = boto3.client('s3')

    for record in event['Records']:
        bucket = record['s3']['bucket']['name']
        key = record['s3']['object']['key']

        if check_if_unencrypted(bucket, key):
            download_path = '/tmp/{}'.format(uuid.uuid4())

            s3.download_file(bucket, key, download_path)
            data = open('test.jpg', 'rb')

            s3.put_object(Bucket=bucket, ServerSideEncryption='AES256', Body=data)

Now that we have out lambda function written we need to create the lambda function inside AWS. The following commands will create the AWS role for Lambda. We first need to create two files. The first is the Trust Policy for the IAM role that will allow Lambda to assume the role.


{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The second file will be the permissions that go along with the role. Note that these permissions give full access to the bucket. Use with caution.


{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::example_bucket"
  },
  {
        "Effect": "Allow",
        "Action": [
          "logs:*"
        ],
        "Resource": "arn:aws:logs:*:*:*"
      }
}

Now that we have those two files, refered from here on as trust.json and permissions.json, we can run the commands to create the role and the lambda function.


aws iam create-role --role-name LambdaRole --assume-role-policy-document file://trust.json

aws iam put-role-policy --role-name LambdaRole --policy-name S3FullAccess --policy-document file://permissions.json

Now that we've created the role for Lambda to use we can create the function. We'll need to ZIP up the code and then upload it for Lambda to run. We will also need to the role ARN from above when we create the function.


zip -9 main.zip main.py

aws lambda create-function \
--region us-east-1 \
--function-name EncryptS3 \
--zip-file fileb://main.zip \
--role arn:aws:iam:us-east-1:123456789012:role:LambdaRole \
--handler main.handler \
--runtime python3.6 \
--profile adminuser \
--timeout 10 \
--memory-size 1024

Next we need to configure both Lambda and S3 to handle notifying Lambda when an object is places in an S3 bucket. We will need another JSON file, policy.json, with the following content that will allow the Lambda Function to access objects in the S3 bucket.


{
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": "arn:aws:lambda:us-east-1:123456789012:function:LambdaRole"
         },
         "Action": [
            "s3:*"
         ],
         "Resource": [
            "arn:aws:s3:::example_bucket/*",
            "arn:aws:s3:::example_bucket"
         ]
      }
   ]
}


aws lambda add-permission \
--function-name EncryptS3 \
--region us-east-1 \
--statement-id some-unique-id \
--action lambda:InvokeFunction \
--principal s3.amazonaws.com \
--source-arn arn:aws:s3:::example_bucket \
--source-account 123456789012 \
--profile adminuser

aws s3api put-bucket-policy --bucket MyBucket --policy file://policy.json

aws iam create-role \
  --role-name InvokeLambdaRole \
  --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "",
          "Effect": "Allow",
          "Principal": {
            "Service": "s3.amazonaws.com"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringLike": {
              "sts:ExternalId": "arn:aws:s3:::*"
            }
          }
        }
      ]
    }'

aws iam put-role-policy \
  --role-name InvokeLambdaRole \
  --policy-name InvokeLambdaPolicy \
  --policy-document '{
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "lambda:InvokeFunction"
         ],
         "Resource": [
           "*"
         ]
       }
     ]
   }'

aws s3api put-bucket-notification \
  --bucket example_bucket \
  --notification-configuration '{
    "CloudFunctionConfiguration": {
      "CloudFunction": "arn:aws:lambda:us-east-1:123456789012:function:LambdaRole",
      "InvocationRole": "arn:aws:iam:us-east-1:123456789012:role:InvokeLambdaRole",
      "Event": "s3:ObjectCreated:*"
    }
  }'

That's everything that's needed. You should be able to upload an object to the S3 bucket and it will be re-uploaded with Server Side Encryption. Go ahead and give it a try and let me know what you think in the comments below. If you have an questions or issues leave a comment or reach out to me on twitter.

Developing Visualization for Security Groups

Wed, 29 Mar 2017 13:02:32 -0400

I was working on a task yesterday and throught I would write it up so that others could possibly benefit from it. I was working to document our AWS enviornment, specifically the security groups around each instance and how the instances are connected to each other and the internet as a whole.

I had been asked several weeks ago if there was some documentation of the AWS environment at work and how instances were interconnected. I didn't have any documentation at the time and it wasn't a huge deal so I let the topic drop. The other day however I was thinking about documenting AWS and that conversion came back into my head. I realized that with the AWS API you could generate the graph of the connections realtivly easily. Since everyone loves pretty pictures I wanted to see if I could visualize it as well.

The first step was to generate the graph of the security groups. This was another chance to use boto3¹, the de-facto Python binding for the AWS API. After toying around I had the code that could generate the graph between each instance and the IPs that were allowed inbound access.


import boto3
import json

def searchIpAddress(ipAddress):
    inputAddress = ipAddress.split('/')[0]
    returnString = ipAddress

    ec2 = boto3.resource('ec2')
    for instance in ec2.instances.all():
        for interface in instance.network_interfaces_attribute:
             if interface['PrivateIpAddress'] == inputAddress:
                 returnString = instance.instance_id
             if 'Association' in interface:
                if interface['Association']['PublicIp'] == inputAddress:
                    returnString = instance.instance_id
    return returnString

ec2 = boto3.resource('ec2')

graph = {}
cache = {}

for instance in ec2.instances.all():
    graph[instance.instance_id] = {}
    for interface in instance.network_interfaces_attribute:
        for group in interface['Groups']:
            security_group = ec2.SecurityGroup(group['GroupId'])
            for permission in security_group.ip_permissions:
                for IP in permission['IpRanges']:
                    if IP['CidrIp'] not in cache.keys():
                        cache[IP['CidrIp']] = searchIpAddress(IP['CidrIp'])
                    source = cache[IP['CidrIp']]
                    if IP['CidrIp'] in graph[instance.instance_id].keys():
                        graph[instance.instance_id][source] = graph[instance.instance_id][source] + 1
                    else:
                        graph[instance.instance_id][source] = 1

nodes = []
links = []

for node in graph:
    nodes.append({'id': node, 'group': 1})
    for edge in graph[node]:
        group = 2
        if edge.split('-')[0] == 'i':
            group = 1
        nodes.append({'id': edge, 'group': group})
        links.append({"source": edge, "target": node, "value": graph[node][edge]})

seen_nodes = set()
nodes_dedup = []
for obj in nodes:
    if obj['id'] not in seen_nodes:
        nodes_dedup.append(obj)
        seen_nodes.add(obj['id'])

data = {'nodes': nodes_dedup, 'links': links}

print(json.dumps(data))

There's one aspect of the code that I want to specifically call out. The method searchIpAddress searches for the instance ID for an IP address. This way if a security group refrences another instance the graph can properly know that. If we were to do this blindly though we would be pulling the list of instances from the API for each IP address for each port. Since we don't want to be wasteful we cache the results and preform a lookup in the cache and only call the method if we've not encountered that IP address before.

Now that we have the graph we need a way to visualize it. Here I wanted to use a Directed Graph² from D3³, the JavaScript visualization library. The python code above will output the graph in a way that can be used by D3. While I was able to get the graph working correctly I felt like the visualization was lacking. The graph couldn't handle the fact that the secutiy groups would have multiple ports vert well. It also couldn't handle the structure of our network very well but that was certainly no fault of the graph.

Overall I'm happy with the exercise. While the visualization aspect didn't turn out how I hoped I was able to get the data and have a way to reproduce it in the future should anyone need it. I hope to incorporate RDS as well in the future but that presents a different set of challenges.

If you've got questions about Amazon Web Services⁴ or cloud technology in general feel free to contact me and I'll see how I can help bring my experience to the problem.

Architecting Serverless Dynamic DNS Using AWS Services

Fri, 24 Jun 2016 16:52:00 -0400

The inspiration for this post and much of its content comes from https://medium.com/aws-activate-startup-blog/building-a-serverless-dynamic-dns-system-with-aws-a32256f0a1d8#.6tzj1o286.

Problem

You've recently set up a server at your home. You don't quite feel comfortable hosting it in a service like AWS or you happened to have a machine lying around you want to try and get some use out of. You've gotten it up and running and forwarded incoming traffic from your router to be forwarded to the server. You set up the DNS and are happy with the results.

Several weeks go by and you're at work. The weather is bad and you find out power was interrupted at your home. You are worried about the server (You didn't use a surge protector or a UPS, did you?) and decide to try and connect. As you fear you can't you go about your work and head home at the end of the day. The weather is clear and you arrive home to find the power is on. You try to connect to your server and everything is fine. You work into the evening and go to bed.

The next day at work you are trying to connect to your server again and find you can't. You try everything but nothing works. You get home and find you can connect fine. You decide to check the external IP address has changed. You call your ISP and find out that they issue dynamic addresses to residential customers and either won't give you a static one or are going to charge you far too much for one.

Solution

After some research you develop a plan to use AWS API Gateway and AWS Lamda. You plan on having a single API endpoint with two modes, get and set. The get method will simply return the IP address of whoever called the API. An example of calling the endpoint in the get mode is as follows:


wget https://....amazonaws.com/prod?mode=get

The return value for the call will be the following:


{
  “return_message”: “176.32.100.36”,
  “return_status”: “success”
}

The client can then use this information to calculate a secure SHA256 hash of the information it needs to pass to the API in the set mode. This hash will consist of IP_AddressHost_NameShared_Secret. If the client wants to update the IP address for host1.dyn.example.com to 192.168.0.1 with the shared secret of P@ssw0rd then it would pass SHA256(192.168.0.1host1.dyn.example.comP@ssw0rd) in the set as show below:


HASH=`echo -n 192.168.0.1host1.dyn.example.comP@ssw0rd | shasum -a 256`

wget https://....amazonaws.com/prod?mode=set&hostname=host1.dyn.example.com&hash=$HASH

If the hostname does not need to be updated the following will be the return value:


{
  “return_message”: “Your IP address matches the current Route53 DNS record.”,
  “return_status”: “success”
}

If the hostname is updated then the following will be the return value:


{
  “return_message”: “Your hostname record host1.dyn.example.com. has been set to 176.32.100.36”,
  “return_status”: “success”
}

You setup the API inside of AWS and configure it to use Lamda as the backend. You create a single Lamda function to use and after some trial and error have the following result:


from __future__ import print_function

import json
import re
import hashlib
import boto3

config_s3_region = 'us-west-2'
config_s3_bucket = 'my_bucket_name'
config_s3_key = 'config.json'

def read_s3_config():
    s3_client = boto3.client(
        's3',
        config_s3_region,
    )

    s3_client.download_file(
        config_s3_bucket,
        config_s3_key,
        '/tmp/%s' % config_s3_key
    )

    full_config = (open('/tmp/%s' % config_s3_key).read())
    return json.loads(full_config)

def route53_client(
  execution_mode,
  aws_region,
  route_53_zone_id,
  route_53_record_name,
  route_53_record_ttl,
  route_53_record_type,
  public_ip
  ):

    route53_client = boto3.client(
        'route53',
        region_name=aws_region
    )

    if execution_mode == 'get_record':
        current_route53_record_set = route53_client.list_resource_record_sets(
            HostedZoneId=route_53_zone_id,
            StartRecordName=route_53_record_name,
            StartRecordType=route_53_record_type,
            MaxItems='2'
        )

        for eachRecord in current_route53_record_set['ResourceRecordSets']:
            if eachRecord['Name'] == route_53_record_name:
                if len(eachRecord['ResourceRecords']) == 1:
                    for eachSubRecord in eachRecord['ResourceRecords']:
                        currentroute53_ip = eachSubRecord['Value']
                        return_status = 'success'
                        return_message = currentroute53_ip
                        return {'return_status': return_status,
                                'return_message': return_message}
                elif len(eachRecord['ResourceRecords']) > 1:
                    return_status = 'fail'
                    return_message = 'You should only have a single value for'\
                    ' your dynamic record.  You currently have more than one.'
                    return {'return_status': return_status,
                            'return_message': return_message}

    if execution_mode == 'set_record':
        change_route53_record_set = route53_client.change_resource_record_sets(
            HostedZoneId=route_53_zone_id,
            ChangeBatch={
                'Changes': [
                    {
                        'Action': 'UPSERT',
                        'ResourceRecordSet': {
                            'Name': route_53_record_name,
                            'Type': route_53_record_type,
                            'TTL': route_53_record_ttl,
                            'ResourceRecords': [
                                {
                                    'Value': public_ip
                                }
                            ]
                        }
                    }
                ]
            }
        )
        return 1

def run_set_mode(set_hostname, validation_hash, source_ip):
    try:
        full_config = read_s3_config()
    except:
        return_status = 'fail'
        return_message = 'There was an issue finding '\
            'or reading the S3 config file.'
        return {'return_status': return_status,
                'return_message': return_message}

    record_config_set = full_config[set_hostname]
    aws_region = record_config_set['aws_region']
    route_53_zone_id = record_config_set['route_53_zone_id']
    route_53_record_ttl = record_config_set['route_53_record_ttl']
    route_53_record_type = record_config_set['route_53_record_type']
    shared_secret = record_config_set['shared_secret']

    if not re.match(r'[0-9a-fA-F]{64}', validation_hash):
        return_status = 'fail'
        return_message = 'You must pass a valid sha256 hash in the '\
            'hash= argument.'
        return {'return_status': return_status,
                'return_message': return_message}


    calculated_hash = hashlib.sha256(source_ip + set_hostname + shared_secret).hexdigest()

    if not calculated_hash == validation_hash:
        return_status = 'fail'
        return_message = 'Validation hashes do not match.'
        return {'return_status': return_status,
                'return_message': return_message}
    else:
        route53_get_response = route53_client(
            'get_record',
            aws_region,
            route_53_zone_id,
            set_hostname,
            route_53_record_ttl,
            route_53_record_type,
            '')

        if not route53_get_response:
            route53_ip = '0'
        elif route53_get_response['return_status'] == 'fail':
            return_status = route53_get_response['return_status']
            return_message = route53_get_response['return_message']
            return {'return_status': return_status,
                    'return_message': return_message}
        else:
            route53_ip = route53_get_response['return_message']

        if route53_ip == source_ip:
            return_status = 'success'
            return_message = 'Your IP address matches '\
                'the current Route53 DNS record.'
            return {'return_status': return_status,
                    'return_message': return_message}
        else:
            return_status = route53_client(
                'set_record',
                aws_region,
                route_53_zone_id,
                set_hostname,
                route_53_record_ttl,
                route_53_record_type,
                source_ip)
            return_status = 'success'
            return_message = 'Your hostname record ' + set_hostname +\
                ' has been set to ' + source_ip
            return {'return_status': return_status,
                    'return_message': return_message}

def lambda_handler(event, context):

    execution_mode = event['execution_mode']
    source_ip = event['source_ip']
    query_string = event['query_string']
    validation_hash = event['validation_hash']
    set_hostname = event['set_hostname']

    execution_modes = ('set', 'get')
    if execution_mode not in execution_modes:
        return_status = 'fail'
        return_message = 'You must pass mode=get or mode=set arguments.'
        return_dict = {'return_status': return_status,
                       'return_message': return_message}

    if execution_mode == 'get':
        return_status = 'success'
        return_message = source_ip
        return_dict = {'return_status': return_status,
                       'return_message': return_message}
    else:
        return_dict = run_set_mode(set_hostname, validation_hash, source_ip)

    return return_dict

An example of the configuration file stored in S3 is as follows:


{
    "host1.dyn.example.com.": {
        "aws_region": "us-west-2",
        "route_53_zone_id": "MY_ZONE_ID",
        "route_53_record_ttl": 60,
        "route_53_record_type": "A",
        "shared_secret": "SHARED_SECRET_1"
    },
    "host2.dyn.example.com.": {
        "aws_region": "us-west-2",
        "route_53_zone_id": "MY_ZONE_ID",
        "route_53_record_ttl": 60,
        "route_53_record_type": "A",
        "shared_secret": "SHARED_SECRET_2"
    }
}

An example of a bash-based client which can be set up as a cron job is as follows:


#!/bin/bash

#./dynamic_dns_lambda_client.sh host1.dyn.example.com. SHARED_SECRET_1 "abc123.execute-api.us-west-2.amazonaws.com/prod"

if [ $# -eq 0 ]
    then
    echo "Usage: $0 host1.dyn.example.com. sharedsecret \"abc123.execute-api.us-west-2.amazonaws.com/prod\""
    exit
fi

myHostname=$1
mySharedSecret=$2
myAPIURL=$3

myIP=`curl -q -s  "https://$myAPIURL?mode=get" | egrep -o '[0-9\.]+'`
myHash=`echo -n $myIP$myHostname$mySharedSecret | shasum -a 256 | awk '{print $1}'`

curl -q -s "https://$myAPIURL?mode=set&hostname=$myHostname&hash=$myHash"
echo

AWS Security Groups and Dynamic IP Addresses

Fri, 10 Jun 2016 09:48:12 -0400

Problem

You are given the task to only allow access to certain AWS resources to the office you work in. You create a Security Group and ask a colleague for the external IP address range assigned to the office. He tells you that there is not static range. The office, along with the rest of the building, share a commercial ISP with dynamic addresses. In addition to that, there is not one but three IPSs that are load balanced for outgoing traffic. The external IP address of the machine you're working on can theoretically change on a per request basis.

Solution

You decide there's nothing you can do about the rest of the building being able to access the resources. It isn't a security threat if they can and there no obvious way that they would be able to find the resources. You decide you'll write a Python script that gets your public IP address. It will then calculate if that address is in the list of address ranges you already know about. If the address is in the range then everything is good. If the address isn't in the range then the script will calculate a new range so you can go update the Security Group.

After an hour or so you have the following script working:


import urllib.request as urllib2
import re
from netaddr import *
import pprint

# Get our external network address and calculate a Class C CIDR based on it.
resource = urllib2.urlopen('http://ipinfo.io/ip')
ext_ip =  resource.read().decode(resource.headers.get_content_charset())
my_ip = re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$",ext_ip).group()
my_cidr = IPNetwork(my_ip)
my_cidr.prefixlen = 24
my_cidr = my_cidr.cidr

print("IP Address: " + str(my_ip))
print("CIDR: " + str(my_cidr))


# The list of existing CIDR ranges
lines = []
ip_list = []
with open('Desktop/AWS/IPs.txt') as infile:
    for line in infile:
        ip_list.append(IPNetwork(line))
        lines.append(line)

# Find the largest ranges that cover our CIDR ranges.
ip_merged = cidr_merge(ip_list)

# Try to find if we are in an existing CIDR by looking at the first address in
# the range.
in_range = False
for network in ip_merged:
    if my_cidr[0] == network[0]:
        in_range = True
        print("Possible match: " + str(network))
        pass
    pass

if in_range:
    print("We should be in the existing range:")
    pass
else:
    print("We might not be in the existing range. Proposed new range:")
    ip_list.append(my_cidr)
    ip_merged = cidr_merge(ip_list)
    pass

lines = []
for network in ip_merged:
    lines.append(str(network))

with open('Desktop/AWS/IPs.txt', 'w') as outfile:
    for line in lines:
        outfile.write(line)

pprint.pprint(ip_merged)

Whenever someone in your office complains that they can't access the resources they need you simple have them run this script. If it states that the existing range should match then you have your colleague run the script several times spaced several minutes apart. This should allow the script to catch the new IP address the ISP is using.

Extra Credit

You decide that having to have your colleagues run this script whenever their work is interrupted is not sufficient. You want to have the script run on a schedule and update the Security Group on it's own. You append the following to the python script to facilitate the automatic updates:

ip_replace_contents = ''

for network in ip_merged:
  ip_replace_contents = ip_replace_contents + "\"" + str(network) "\", "
  pass
ip_replace_contents = ip_replace_contents.rstrip(", ")

replacements = {'IP_REPLACE_CONTENTS':ip_replace_contents}

with open('Desktop/Terraform/Security_Groups.tf') as infile, open('Desktop/Terraform/' + datetime.datetime.now().strftime("%Y-%m-%d") + '/Security_Groups.tf', 'w') as outfile:
    for line in infile:
        for src, target in replacements.iteritems():
            line = line.replace(src, target)
        outfile.write(line)

You schedule the following shell script to run every week day at 6:00 AM to update the Security Group before the work day begins:

#!/bin/bash

NOW=$(date +"%Y-%m-%d")

python dynamic_ips.py
terraform apply Desktop/Terraform/$NOW

Everyone in the office is happy knowing that the resources on AWS are safe and they don't have to worry about not being able to access them themselves.

Hello, World

Wed, 08 Jun 2016 14:23:02 -0400

Hello, World